{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Welcome to PixieDust\n",
    "\n",
    "This notebook features an introduction to PixieDust, the Python library that makes data visualization easy. \n",
    "\n",
    "## Get started\n",
    "\n",
    "This notebook is pretty simple and self-explanatory, but it wouldn't hurt to load up the [PixieDust documentation](https://pixiedust.github.io/pixiedust/) so you have it. \n",
    "\n",
    "New to notebooks? Don't worry, all you need to know to use this notebook is that to run code cells, put your cursor in the cell and press **Shift + Enter**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Make sure you have the latest version of PixieDust installed on your system\n",
    "# Only run this cell if you did _not_ install PixieDust from source\n",
    "# To confirm you have the latest, uncomment the next line and run this cell\n",
    "#!pip install --user --upgrade pixiedust"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have PixieDust installed and up-to-date on your system, you need to import it into this notebook. This is the last dependency before you can play with PixieDust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pixiedust database opened successfully\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <div style=\"margin:10px\">\n",
       "            <a href=\"https://github.com/ibm-watson-data-lab/pixiedust\" target=\"_new\">\n",
       "                <img src=\"https://github.com/ibm-watson-data-lab/pixiedust/raw/master/docs/_static/pd_icon32.png\" style=\"float:left;margin-right:10px\"/>\n",
       "            </a>\n",
       "            <span>Pixiedust version 1.0.4</span>\n",
       "        </div>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "ImportError",
     "evalue": "No module named pyspark",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-c4bb40302b4e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Run this cell\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mpixiedust\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/mbrobergus.ibm.com/Documents/pixiedust/pixiedust/__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m     \u001b[0;31m#shortcut to packageManager\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m     \u001b[0;32mimport\u001b[0m \u001b[0mpixiedust\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpackageManager\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpackageManager\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m     \u001b[0mprintAllPackages\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpackageManager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprintAllPackages\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m     \u001b[0minstallPackage\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpackageManager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minstallPackage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/mbrobergus.ibm.com/Documents/pixiedust/pixiedust/packageManager/__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;31m# -------------------------------------------------------------------------------\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mpackageManager\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPackageManager\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;31m#shortcut to packageManager\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/mbrobergus.ibm.com/Documents/pixiedust/pixiedust/packageManager/packageManager.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpixiedust\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstorage\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpixiedust\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprintEx\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 25\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpyspark\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSparkContext\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     26\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mpackage\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPackage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mdownloader\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDownloader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mRequestException\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mImportError\u001b[0m: No module named pyspark"
     ]
    }
   ],
   "source": [
    "# Run this cell\n",
    "import pixiedust"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you see the success message output from running `import pixiedust`, you're all set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Behold, display()\n",
    "\n",
    "In the next cell, build a very simple dataset and store it in a variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'SQLContext' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-e7413d831fe3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Run this cell to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# a) build a SQL context for a Spark dataframe\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0msqlContext\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mSQLContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;31m# b) create Spark dataframe, and assign it to a variable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m df = sqlContext.createDataFrame(\n",
      "\u001b[0;31mNameError\u001b[0m: name 'SQLContext' is not defined"
     ]
    }
   ],
   "source": [
    "# Run this cell to\n",
    "# a) build a SQL context for a Spark dataframe \n",
    "sqlContext=SQLContext(sc) \n",
    "# b) create Spark dataframe, and assign it to a variable\n",
    "df = sqlContext.createDataFrame(\n",
    "[(\"Green\", 75),\n",
    " (\"Blue\", 25)],\n",
    "[\"Colors\",\"%\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data in the variable we just created is ready to be displayed, without any code other than the call to `display()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "handlerId": "pieChart",
      "keyFields": "Colors",
      "rowCount": "100",
      "title": "Colors in this pie chart, by %",
      "valueFields": "%"
     }
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.pd_warning{display:none;}</style><div class=\"pd_warning\"><em>Hey, there's something awesome here! To see it, open this notebook outside GitHub, in a viewer like Jupyter</em></div>\n",
       "        <div class=\"pd_save is-viewer-good\" style=\"padding-right:10px;text-align: center;line-height:initial !important;font-size: xx-large;font-weight: 500;color: coral;\">\n",
       "            Colors in this pie chart, by %\n",
       "        </div>\n",
       "    <div id=\"chartFigureb4be9c64\" class=\"pd_save is-viewer-good\" style=\"overflow-x:auto\">\n",
       "            \n",
       "                    \n",
       "                            <center><img style=\"max-width:initial !important\" src=\"\" class=\"pd_save\"></center>\n",
       "                        \n",
       "                    \n",
       "                \n",
       "        </div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run this cell to display the dataframe above as a pie chart\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the cell above, you should have seen a Spark dataframe displayed as a **pie chart**, along with some controls to tweak the display. All that came from passing the dataframe variable to `display()`.\n",
    "\n",
    "In the next cell, we'll pass more interesting data to `display()`, which will also offer more advanced controls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "clusterby": "year",
      "handlerId": "barChart",
      "keyFields": "category",
      "rowCount": "100",
      "title": "Customers by Category clustered by Year",
      "valueFields": "unique_customers"
     }
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.pd_warning{display:none;}</style><div class=\"pd_warning\"><em>Hey, there's something awesome here! To see it, open this notebook outside GitHub, in a viewer like Jupyter</em></div>\n",
       "        <div class=\"pd_save is-viewer-good\" style=\"padding-right:10px;text-align: center;line-height:initial !important;font-size: xx-large;font-weight: 500;color: coral;\">\n",
       "            Customers by Category clustered by Year\n",
       "        </div>\n",
       "    <div id=\"chartFigure9d89b30a\" class=\"pd_save is-viewer-good\" style=\"overflow-x:auto\">\n",
       "            \n",
       "                    \n",
       "                            <center><img style=\"max-width:initial !important\" src=\"\" class=\"pd_save\"></center>\n",
       "                        \n",
       "                    \n",
       "                \n",
       "        </div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create another dataframe, in a new variable\n",
    "df2 = sqlContext.createDataFrame(\n",
    "[(2010, 'Camping Equipment', 3),\n",
    " (2010, 'Golf Equipment', 1),\n",
    " (2010, 'Mountaineering Equipment', 1),\n",
    " (2010, 'Outdoor Protection', 2),\n",
    " (2010, 'Personal Accessories', 2),\n",
    " (2011, 'Camping Equipment', 4),\n",
    " (2011, 'Golf Equipment', 5),\n",
    " (2011, 'Mountaineering Equipment',2),\n",
    " (2011, 'Outdoor Protection', 4),\n",
    " (2011, 'Personal Accessories', 2),\n",
    " (2012, 'Camping Equipment', 5),\n",
    " (2012, 'Golf Equipment', 5),\n",
    " (2012, 'Mountaineering Equipment', 3),\n",
    " (2012, 'Outdoor Protection', 5),\n",
    " (2012, 'Personal Accessories', 3),\n",
    " (2013, 'Camping Equipment', 8),\n",
    " (2013, 'Golf Equipment', 5),\n",
    " (2013, 'Mountaineering Equipment', 3),\n",
    " (2013, 'Outdoor Protection', 8),\n",
    " (2013, 'Personal Accessories', 4)],\n",
    "[\"year\",\"category\",\"unique_customers\"])\n",
    "\n",
    "# This time, we've combined the dataframe and display() call in the same cell\n",
    "# Run this cell \n",
    "display(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## display() controls\n",
    "\n",
    "### Renderers\n",
    "This chart like the first one is rendered by matplotlib. With PixieDust, you have other options. To toggle between renderers, use the `Renderers` control at top right of the display output:\n",
    "1. [Bokeh](http://bokeh.pydata.org/en/0.10.0/index.html) is interactive; play with the controls along the top of the chart, e.g., zoom, save\n",
    "1. [Matplotlib](http://matplotlib.org/) is static; you can save the image as a PNG\n",
    "\n",
    "### Chart options\n",
    "\n",
    "1. **Chart types**: At top left, you should see an option to display the dataframe as a table. You should also see a dropdown menu with other chart options, including bar charts, pie charts, scatter plots, and so on.\n",
    "1. **Options**: Click the `Options` button to explore other display configurations; e.g., clustering\n",
    "\n",
    "To know more : https://pixiedust.github.io/pixiedust/displayapi.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading External Data\n",
    "So far, we've worked with data hard-coded into our notebook. Now, let's load external data (CSV) from an addressable `URL`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "pixiedust": {
     "displayParams": {
      "color": "origin",
      "handlerId": "scatterPlot",
      "keyFields": "horsepower",
      "kind": "hex",
      "rendererId": "matplotlib",
      "rowCount": "1000",
      "title": "Distribution of MPG per Horsepower",
      "valueFields": "mpg"
     }
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.pd_warning{display:none;}</style><div class=\"pd_warning\"><em>Hey, there's something awesome here! To see it, open this notebook outside GitHub, in a viewer like Jupyter</em></div>\n",
       "        <div class=\"pd_save is-viewer-good\" style=\"padding-right:10px;text-align: center;line-height:initial !important;font-size: xx-large;font-weight: 500;color: coral;\">\n",
       "            Distribution of MPG per Horsepower\n",
       "        </div>\n",
       "    <div id=\"chartFigurec4913c69\" class=\"pd_save is-viewer-good\" style=\"overflow-x:auto\">\n",
       "            \n",
       "                    \n",
       "                            <center><img style=\"max-width:initial !important\" src=\"\" class=\"pd_save\"></center>\n",
       "                        \n",
       "                    \n",
       "                \n",
       "        </div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load a CSV with pixiedust.sampledata()\n",
    "df3 = pixiedust.sampleData(\"https://github.com/ibm-watson-data-lab/open-data/raw/master/cars/cars.csv\")\n",
    "display(df3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see a scatterplot above, rendered again by matplotlib. Look at the `Renderer` menu at top right. You should see options for **Bokeh** and now, **Seaborn**. If you don't see Seaborn, it's not installed on your system. No problem, just install it by running the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# To install Seaborn, uncomment the next line, and then run this cell\n",
    "#!pip install --user seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*If you installed Seaborn, you'll need to also restart your notebook kernel, and run the cell to `import pixiedust` again. Find **Restart** in the **Kernel** menu above.*"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
